New snippet for jpeg/zip: 4tu-60cdce27-liposome-yolo-boxes-confocal - #62

snippets: round 1 pushed 16fe7340f499 to snippets/issue-55 · $3.42

Added the snippet jpeg-zip-glob with its example binding for 4tu-60cdce27-liposome-yolo-boxes-confocal, committed as 16fe734 (one new directory, nothing else touched, not pushed). The harness smoke on the real data passes.

What it does

• Reads: JPEG images chosen by glob from the members of a zip, one image per z plane, decoded with imagecodecs. The zip can be a plain URL or a member of another zip, which is this dataset's case.
• glob (required): an fnmatch pattern over the inner zip's member names, e.g. val/*.jpg.
• samples (optional): which RGB samples become the c axis, e.g. rg; one letter gives a single plane with no c axis.
• Requires: numpy and imagecodecs only, both on the allowlist.

Departures from the issue's draft

• URL form: each URL names the inner zip (<4TU zip>::<…>/obj.zip) and the image pattern goes in params.glob, not the draft's <zip>::<inner zip>::<glob>. harness fetch and harness stage call io.expand and io.fetch on every binding URL and only resolve one :: level, so the triple form would fail there. The snippet rejects it with that explanation.
• Whole set bound: four raws, one per inner zip (560 + 140 + 32 + 8 = 740 images), each zcyx (N, 2, 512, 512) uint8, instead of the 8-image sample.
• Anchored globs: obj/*.jpg and val/*.jpg, because the inner zips also hold __MACOSX/<folder>/._*.jpg sidecars that a bare *.jpg matches and that are not JPEGs.
• samples: rg: I decoded all 740 images. Blue is residue (per-set mean 0.3–1.5 of 255, max 4 in the 2-class set and 56–64 in the 1-class set) while red and green reach 255, so blue is dropped.

Left for a human

• Which of red and green is eGFP-MinC and which is DOPE-Cy5 is not stated in the record or the files; the binding's reasons say so.
• No labels are bound. The YOLO box tables have no representation in the binding format (segmentation_type / label_key), as the issue notes.
• Voxel size is None; the record's 250 nm comes from the paper.

Testing

Smoke summary, for the PR description (harness smoke examples/4tu-60cdce27-liposome-yolo-boxes-confocal.yaml, host data.4tu.nl, 5 GB cap):

{"ok": true,
 "shapes": {"raw": [[560, 2, 512, 512], [140, 2, 512, 512], [32, 2, 512, 512], [8, 2, 512, 512]], "labels": []},
 "warnings": [], "errors": [], "bytes_fetched": 103390555}

• Record file: the record.yaml I ran against was a cut-down copy of the issue's Record block, keeping only technical.arrays (the part smoke reads), not the full block.
• Crop and verify: harness crop --n 4 made 4 crops with no errors, but harness verify rejected all 4 with "crop.json has no licence", because the issue's Record block has no license field. With a placeholder licence added to my local record copy, a second run made 6 crops and verify passed 6/6. One crop viewed by eye shows liposomes.
• Fetch: harness fetch on the binding reports ok with all four inner zips.
• Repo suites: harness tests 225 passed, CI-checker tests 21 passed, and ci/check_pr.py --author 'mia-autodev[bot]' against the base commit passes. The checkout has no pixi environment, so I ran these with the pinned harness environment's Python rather than pixi run test.
• Snippet tests: 8 ad-hoc tests over the harness's local range server pass, but they are not committed because a snippet PR may not add files under harness/tests. They cover stored and deflated nesting, a plain zip URL, grey JPEGs, samples, an outer glob, several zips, region indexing and the error paths.

Review judge: approve

The PR adds a new snippet, jpeg-zip-glob, that decodes JPEG images selected by glob from a zip, including a zip nested inside the dataset's zip. Each image becomes one z plane and the chosen RGB samples become c. The example binding reads all four inner zips (740 images) with red and green as channels and binds no labels, as the issue requires. Only additions under one new snippet directory, dependencies are on the allowlist, and nothing serious blocks it.

confidence 0.77 · model claude-opus-5-5 · prompt f5f9c39cc98e · policy v1 · $0.26

Shadow mode: this verdict is recorded and compared with the human label; it decides nothing.

Crops at 16fe7340f499cbc4d8d70e8af069520fe7711d7e: passed, 16 of 16 verified (at least 8 needed); the crop job fetched 103390555 bytes.

16 crops

raw	overlay	caption
		i27352958-e1794578557165-b0-000 · z=112 c=0 · (177, 22)
		i27352958-e1794578557165-b0-001 · z=112 c=0 · (91, 219)
		i27352958-e1794578557165-b0-002 · z=107 c=0 · (228, 198)
		i27352958-e1794578557165-b0-003 · z=29 c=0 · (45, 239)
		i27352958-e1794578557165-b0-004 · z=29 c=0 · (237, 194)
		i27352958-e1794578557165-b0-005 · z=111 c=0 · (70, 19)
		i27352958-e1794578557165-b0-006 · z=111 c=0 · (107, 224)
		i27352958-e1794578557165-b0-007 · z=188 c=1 · (196, 245)
		i27352958-e1794578557165-b0-008 · z=188 c=1 · (105, 82)
		i27352958-e1794578557165-b0-009 · z=1 c=1 · (132, 181)
		i27352958-e1794578557165-b0-010 · z=38 c=0 · (171, 167)
		i27352958-e1794578557165-b0-011 · z=332 c=1 · (27, 79)
		i27352958-e1794578557165-b0-012 · z=332 c=1 · (227, 134)
		i27352958-e1794578557165-b0-013 · z=463 c=1 · (131, 14)
		i27352958-e1794578557165-b0-014 · z=39 c=0 · (74, 177)
		i27352958-e1794578557165-b0-015 · z=39 c=0 · (178, 4)

Binding (sha256 2cd209a527fabdfa1745437628da232316a6b7f8d947b455a46d8110b7ff1053):

record: 4tu-60cdce27-liposome-yolo-boxes-confocal
record_commit: 3f75709f1d2778fe86fd9c436b2b5ff948a72c7b
snippets_commit: 16fe7340f499cbc4d8d70e8af069520fe7711d7e
raw:
- snippet: jpeg-zip-glob
  urls:
  - https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data
    - YOLO training and testing/Fig3 - FigS4/MinD 1 class/20240108_MinD_1class_training
    data and results/obj.zip
  params:
    glob: obj/*.jpg
    samples: rg
- snippet: jpeg-zip-glob
  urls:
  - https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data
    - YOLO training and testing/Fig3 - FigS4/MinD 1 class/20240108_MinD_1class_training
    data and results/val.zip
  params:
    glob: val/*.jpg
    samples: rg
- snippet: jpeg-zip-glob
  urls:
  - https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data
    - YOLO training and testing/Fig3 - FigS4/MinCDE 2 classes/20241107_MinCDE_2classes_training
    data and results/obj.zip
  params:
    glob: obj/*.jpg
    samples: rg
- snippet: jpeg-zip-glob
  urls:
  - https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data
    - YOLO training and testing/Fig3 - FigS4/MinCDE 2 classes/20241107_MinCDE_2classes_training
    data and results/val.zip
  params:
    glob: val/*.jpg
    samples: rg
labels: []
reasons:
- 'jpeg-zip-glob: the images are JPEGs inside obj.zip / val.zip, which are themselves
  deflated members of the record''s one zip; no other snippet decodes JPEG or looks
  inside a nested zip'
- 'each url is the inner zip as a member of the 4TU zip (`<zip>::<...>/obj.zip`),
  not the draft''s `<zip>::<inner zip>::<glob>`: the harness''s own fetch and stage
  call io.expand and io.fetch on every binding url, which resolve one `::` level,
  so the url names a file the harness can fetch and cache (the inner zip) and the
  pattern for the images inside it is params.glob'
- four raws, one per inner zip, so the whole set is bound (560 + 140 + 32 + 8 = 740
  images, the record's n_items) rather than the 8-image sample, and the 1-class /
  2-class sets and their obj (training) / val splits stay separate volumes; each is
  a z stack of independent 512x512 fields in member-name order, not a 3D volume (record
  dimensionality 2D)
- 'glob obj/*.jpg and val/*.jpg, anchored at the folder: each inner zip also holds
  macOS sidecars __MACOSX/<folder>/._<name>.jpg (AppleDouble, not JPEG) that a bare
  *.jpg would match, and the .txt box files'
- 'samples rg: the record says the two confocal channels look stored as R and G with
  B near 0; decoding all 740 images confirmed it (per-set B mean 0.3-1.5 of 255, B
  max 4 in the 2-class set and 56-64 in the 1-class set, against R and G reaching
  255), so B is JPEG chroma residue and is dropped; c 0 = red sample, c 1 = green
  sample. Which of them is eGFP-MinC and which DOPE-Cy5 (or the 1-class set''s stains)
  is not stated in the record or the files'
- 'no labels bound: the annotations are YOLO box tables (record encoding boxes-table,
  one .txt per jpg), which have no segmentation_type or label_key kind in the binding
  format; a human has to decide how boxes are represented before a snippet can read
  them'
- 'voxel size: the snippet reports None (a JPEG carries no pixel size); the record''s
  250 nm x/y comes from the paper'

/approve 16fe734